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Abstract 

We present a search for charged Higgs bosons in decays of pair-produced 
top quarks using 109.2 ± 5.8 pb _1 of data recorded from pp collisions at 
^/s = 1.8 TeV by the D0 detector during 1992-96 at the Fermilab Tevatron. 
No evidence is found for charged Higgs production, and most parts of the 
[Mj^+,tan/3] parameter space where the decay t — ► H + b has a branching frac- 
tion close to or larger than that for t — ► W + b are excluded at 95% confidence 
level. Assuming nit = 175 GeV and o~(pp — ► tt) = 5.5 pb, for M H += 60 GeV, 
we exclude tan/3 < 0.97 and tan/3 > 40.9. 



The Higgs sector of the standard model (SM) consists of a single complex doublet scalar 
field responsible for breaking electroweak symmetry and generating gauge boson masses. The 
simplest extension of the Higgs sector to two complex doublets appears in many theories 
beyond the SM, including supersymmetry (SUSY). Our study is based on the two-Higgs- 
doublet model where one doublet couples to up-type quarks and neutrinos, and the other 
couples to down- type quarks and charged leptons, as required by SUSY [|IJ. Under these 
circumstances, electroweak symmetry breaking leads to five physical Higgs bosons: two 
neutral scalars h° and H°, a neutral pseudoscalar A , and a pair of charged scalars H ± . The 
extended Higgs sector has two new parameters: M#+ and tan/3, where tan/3 is defined as 
the ratio of the vacuum expectation values of the two Higgs fields. 

Direct searches for e + e~ — > H + H~X at LEP have set lower limits of 57.5-59.5 GeV 
on Mh+ at the 95% confidence level (CL) irrespective of tan/3 0. A measurement of the 
inclusive b — > S7 decay rate gives CLEO an indirect limit of Mu+ > [(244 + 63/(tan/3) L3 ] 
GeV, assuming only a two-Higgs-doublet extension to the SM j3|. From a measurement 
of the b — > tvX branching fraction, ALEPH constrains tan /3/M H + < 0.52 GeV -1 at 90% 
CL [|J. Based on a search for charged Higgs in decays of pair-produced top quarks using 
hadronic decays of the r lepton, CDF has published limits in the [M#+ ,tan /3] parameter 
space for tan (3 > 5 M . Our search, also for H ± in decays of tt, covers the entire range of 
tan /3 in which leading order perturbative calculations are valid. 

At leading order, the H + coupling to a down-type (up-type) quark or neutral (charged) 
lepton is proportional to the fermion mass multiplied by tan/3 (cot/?). The SM requires a 
t quark to decay almost exclusively to a W boson and a b quark, i.e., B(t — > W + b) « 1. 
However, if H^ exist with M H + < m t — mb, and tan/? is either very large or very small, 
then B(t — >■ H + b) can be significant. We assume B(t — > H + b) + B(t — > W + b) = 1. For 
any given tan/?, B(t — > H + b) decreases as M H + increases. It is further assumed that M s o 
(S° = h°, H°, or A ) are large enough for the decays H + — > S°W + to be highly suppressed 
for real or virtual S° and W + bosons. Decays H + — > U°IU + , where V° = 7 or Z, are absent 
at the tree level ||. Hence, H + can only decay to fermion-antifermion pairs. Consequently, 
if Mh+ < m t — 777.6, one might expect H + — > r + z/ (favored if tan/3 is large) and H + — > cs 
(favored if tan/5 is small) to be the only significant possibilities. Indeed, B(H + — ► t + v) rs 1 
if tan/3 > 10. But if tan/3 < 2 and M^+ > 130 GeV, then the large mass of the t quark 
causes B(H+ -► t*b -► W + bb) to exceed B(H + ^ cs) 0. 

Figure |l] shows the region of the [M#+,tan/3] plane examined in this analysis. The lower 
and upper boundaries on tan/3 (0.3, 150) are required for the applicability of perturbative 
calculations in H + Yukawa coupling to t and b quarks. The minimum for M#+ is chosen at 
50 GeV, somewhat below the most recent lower limits from LEP. This search is restricted 
to M H + < 160 GeV, somewhat less than m t — m^ (assuming m t = 175 GeV); otherwise, the 
width of the charged Higgs T(H + ) becomes too large (> 7.5 GeV) near the upper boundary 
on tan/3, and leading-order calculations become unreliable. For the same reason, T(t) is 
required to be < 15 GeV. Since T{t — ► W + b) w 1.5 GeV, irrespective of [M#+,tan/3], this 
amounts to requiring B(t —* H + b) < 0.9, and thereby excludes from our analysis the dark- 
shaded regions at the two bottom corners of Fig. |l[ The cross-hatched regions correspond 
to B(t — > H + b) > 0.5. Also shown in Fig. [I] are the decay modes of H + that dominate 
in different parts of the parameter space. Analogous charge-conjugate expressions hold for 
H-. 
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FIG. 1. The parameter space explored in this analysis. Regions where B(t — ► H + b) > 0.5 are 
shown cross-hatched, with the labels for various decay modes of the charged Higgs indicating their 
regions of dominance. Regions where B(t — ► H + b) > 0.9 (dark shaded areas) are not considered. 



TABLE I. The Z+jets and Z+jets//i event selection criteria. 





/+jets 


l+jets/fj, 


Pt(1) 


>20 GeV 


>20 GeV 


k( M )l 


<2.0 (1.7) 


<2.0 (1.7) 


$T 


>25 GeV 


>20 GeV 


E T (j) 


>15 GeV 


>20 GeV 


\m\ 


<2.0 


<2.0 


# of jets (rij) 


> 4 


> 3 


# of //-tags 





> 1 


Aplanarity 


>0.065 


>0.040 


H T ^EZiE T (ji) 


>180 GeV 


>110 GeV 


p T (l) + $ T 


>60 GeV 


- 


\v(w)\ 


<2.0 


- 



For each top quark, there are four possible decay modes whose branching fractions depend 
on M H+ and tan/5: (1) t -> W+b; (2) t -»■ #+&, #+ -► cs; (3) £ -^ H+b, H+ -> 1^+66; 
and (4) £ — >• iJ + 6, H + — > r + z/. If the decay mode of £ (t) is denoted by i (j), then the total 
acceptance for any set of selection criteria is given by 



A(M H+1 tan/5) = J2 t iyj {M H +)B l (M H+1 tim(3)B j (M H +, tan/?), 



(1) 



*J=i 



where e^ is the efficiency for channel {i,j}, and BjBj is the branching fraction. All Bi 
depend strongly on both M H + and tan/5; e^i depends on neither, and all other e it j depend 
on Mh+, but not on tan/5. 

A strong dependence of signal characteristics on the parameters of the model makes an 
appearance search for signal difficult for us. We therefore perform a disappearance search 
using selection criteria optimized for the SM channel {1,1}. One expects the efficiencies of 
these criteria for channels involving t — > H + b decays to be substantially different from that 
for channel {1,1}- Consequently, if the assumption of B\ = 1 leads to a measurement of the 
top quark pair production cross section a(ti) in good agreement with theoretical predictions, 
then those regions of the [Mh+ ,tan /5] parameter space where Bi is sufficiently large for any 
i 7^ 1 can be excluded. This strategy serves us well for i = 2 and 4, but not for i = 3. 

The D0 detector is described in Ref. 0. We use the same reconstruction algorithms for 
jets, muons, and electrons as used in our previous top quark analyses, and the same event 
selection criteria as for the measurement of a(tt) in lepton+jets final states 0. These criteria 
are optimized for it events where both top quarks decay to Wb, with one W decaying into ev 
or fiu, and the other into a qq' pair. The final state in such events is characterized by a high- 
Pt isolated lepton, large missing transverse energy ($ T ), and four jets. The main sources 
of background are IV+jets events and QCD multijet events with a misidentified lepton and 
large $ T . Two of the jets in signal events are initiated by b quarks. A b jet can be tagged 
by a muon contained within the jet (eB p» 0.2 per it event). Since such tagging is unlikely 
in background events, other requirements can be less restrictive for an event containing a 
//-tagged jet. This class of /i-tagged events is denoted by /+jets//i. Events without a \i- 
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TABLE II. The integrated luminosity, the number of observed events, and the expectations 
from background and SM it signal (assuming m t = 175 GeV, a (it) = 5.5 pb), for Z+jets and 
l+jets/fi selections combined. 



Integrated luminosity, C 


109.2 ± 5.8 pb 


Estimated background, n B 


11.2 ±2.0 


Expected signal (SM), ns 


19.7 ±3.5 


Total events expected (SM) 


30.9 ± 4.0 


Events observed, n b s 


30 



tagged jet, denoted by Z+jets, are subject to stricter requirements on kinematics. Details of 
the selection criteria, summarized in Table |, can be found in Ref. f9j. For m t = 175 GeV, 
the selection efficiency for it -> W + bW~b events is 3.42±0.11(stat)±0.55(syst) %. The jet 
energy scale, particle identification, and modeling of the signal are the primary sources of 
systematic uncertainty. The integrated luminosity, the number of observed events, and the 
expected it signal (assuming B(t — * W + b) = 1) and background are given in Table [H[ 

The measured values of a (it) P,|H| and m t [M are based on the assumption of B(t — > 
W + b) = 1, and cannot be used in this analysis. Hence, in our search, a (it) and m t serve 
as input parameters. Production of it takes place primarily via strong interactions, and the 
cross section is not affected by the existence of H ± (assuming no contribution from SUSY 
processes). Calculations of a (it) based on QCD should therefore be reasonable fl^-|l4"l. 
While there is no strong reason to use the measured value of mt when allowing B(t — > 
H + b) to be large, lacking a compelling argument in favor of an alternative choice, we use 
mt = 175 GeV. A special version of isajet |15| that includes the process H + — > W + bb is 



used for Monte Carlo simulation of it events, and a similarly modified version of pythia JT6 
for verification of the efficiencies. 

Table |H] shows that the hypothesis of B\ « 1 agrees well with our experimental result. 
Using Monte Carlo samples, the efficiencies and corresponding uncertainties are calculated at 
several values of M#+ , and parametrized for each channel. The efficiencies for all channels, 
for Mh+ = 125 GeV, are listed in Table |T|. The dependence of efficiency on M B + varies from 
channel to channel, but efficiencies for a given channel rarely differ by more than a factor of 
two over the range of M H + considered. While e 2j 2 is practically zero, ei )3 and e 3)3 are close to 
ei 5 i. Consequently, we can exclude at a high level of confidence those regions of parameter 
space where B2 ~ 1 (small tan/3, small M#+), because, with almost no observable signal, 
it is extremely unlikely that an expected background of 11.2 ± 2.0 events fluctuated to the 
observed 30. However, in regions where B% is comparable to or larger than B\ (small tan/?, 
large M#-+), the expected number of events is about the same as that observed, and therefore 
such regions cannot be excluded. Low efficiencies for it decays involving H + — ► t + v helps 
exclude regions where B4 is large (large tan/3). 

For n bs observed events, the joint posterior probability density for M H + and tan/3 is 
given by 

P(M H+ , tan P\n ohs ) oc I G(C) I G(n B ) f G(A)P(n ohs \fi)dAdn B dC, (2) 

where P(n b s |/x), is the Poisson probability of observing n b s events, given a total (signal + 



TABLE III. The efficiencies ejj of our selection criteria (in %), for m* = 175 GeV and 
Mjj+ = 125 GeV, for various decay modes of it. The row indices (i) denote: (1) t — > W + b; 
(2) t -> F+6, F+ -> cs; (3) t -► #+6, #+ -> IT+66; and (4) t -> F+6, F+ -> r+i/. The respective 
charge conjugate decays are denoted by the column indices (j). 



1 



3.42 ±0.56 
2.23 ±0.37 

3.35 ±0.61 

1.36 ±0.25 



2.23 ±0.37 
0.04 ±0.01 
2.21 ±0.37 
1.07 ±0.20 



3.35 ±0.61 

2.21 ±0.37 
3.71 ±0.67 
1.74 ±0.36 



1.36 ±0.25 
1.07 ±0.20 
1.74 ±0.36 
0.41 ±0.09 



background) expectation of 

H(M H +, tan 0) = A{M H +, tan (3)a{tt)C + n B , 



(3) 



and G represents a Gaussian distribution. The means and widths of the Gaussians for the in- 
tegrated luminosity C, and the number of background events Ub, are given in Table O, while 
those for the acceptance A(M#+,tan/3), are calculated using Eq. ([J), with parametrized 
functions for ey, and leading order calculations of Bi, Bj. 

Equation (fj), which we parametrize as a function of M H + and tan/?, gives a Bayesian 
posterior probability density for those parameters [17] . The prior distribution is assumed to 
be uniform in M^+ and in log 10 (tan/3). Assuming instead that the prior is uniform in M^+ 
and in B{H + — > t + u) does not significantly alter the posterior distribution. To calculate 
probabilities, a Monte Carlo integration is carried out by spanning the parameter space in 
steps of 5 GeV in M H + from 50 GeV to 160 GeV, with 25 uniform steps in log 10 (tan/3) 
covering the range 0.3 < tan/? < 150 at each value of Mh+, and performing 200,000 trials 
of Eq. (H) at each step. The predicted probability for observing n b s events, evaluated at 
M H + = 80 GeV, for different values of tan/3, is shown in Fig. ||(a), while Fig. ||(b) shows 
the posterior probability density for tan/3 corresponding to n b s = 30, for M H + = 80 GeV. 
The 95% CL exclusion boundary in the [M#+,tan/3] plane is obtained by integrating the 
probability density P(M H +, tan /?|n b s ), given by Eq. (g), between contours of constant P. 
The results, corresponding to m t = 175 GeV, are shown in Fig. |3| for three values of a {it). 
The largest value of a {it) (5.5 pb, with QCD resummation scale set to m t [0) yields the 
most conservative limits. Tighter limits are set for smaller values of cr{it), such as those given 
in Refs. []1~3" , |T4" |. Figure |3| also shows the result of a frequentist analysis of our data wherein 
a point in the parameter space is excluded if more than 95% of the trials of Eq. (|2|) at that 
point yield n b s < 30. Due caution must be exercised in comparing Bayesian and frequentist 
results since the interpretation of "confidence level" is different between the two. For a given 
value of &{it), the excluded region increases with increasing m t within the range 170 GeV 
< m t < 180 GeV, by an extent comparable to that from a similar fractional decrease in 
a {it) at a fixed m t . 

To summarize, in a search for a charged Higgs boson that considers all of its fermionic 
decay modes, we find no evidence of signal in the region of M H + < 160 GeV, improve 
previous limits in the region of large tan/3, and exclude a significant part of the previously 
unexplored region of small tan/3. Assuming m t = 175 GeV and a{it) = 5.5 pb, tan/3 < 0.97 
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FIG. 2. (a) Distribution of the number of Monte Carlo experiments in the n b s vs. log 10 (tan/?) 
plane for m t = 175 GeV, a(tt) = 5.5 pb, and M H + = 80 GeV. (b) Posterior probabilty density for 
tan/3, given the experimentally observed value of n Q bs = 30 (the slice shown in (a)), for the above 
parameters. 
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FIG. 3. The 95% CL exclusion boundaries in the [M#+,tan/3] plane for m t = 175 GeV, and 
value of a (it) set to 5.5 pb (hatched area, solid lines), 5.0 pb (dashed lines), and 4.5 pb (dotted 
lines). The thicker dot-dashed lines inside the hatched area represent the exclusion boundaries 
obtained from a frequentist analysis with a (it) = 5.5 pb. 

and tan/3 > 40.9 are excluded at 95% CL for M H += 60 GeV. The limits become less 
stringent with increasing Mh+. Within the range of 0.3 < tan/? < 150, no lower limit can 
be set on tan/? for M#+ > 124 GeV, and no upper limit for M#+ > 153 GeV. A comparison 
between Figs. [I] and [3] shows that all regions of the [M H +, tan j3] parameter space where 
B(t —> H + b) > 0.45, except where B(H + —> W + bb) is large, are excluded at 95% CL. 
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